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1  Introduction 


Associative  memories  are  one  of  the  most  interesting  applications  of  neural  networks. 
In  general,  an  associative  memory  stores  a  set  of  information,  called  memories.  The  in¬ 
formation  is  stored  in  a  format  such  that  when  an  external  stimulus  is  presented  into  the 
system,  the  system  evolves  to  a  stable  state  that  is  closest  to  the  input  data.  We  can  view 
this  process  as  a  content-addressable  memory  since  the  stored  memory  is  retrieved  by  the 
contents  of  the  input  and  not  by  the  specific  address.  In  other  words,  the  memory  can 
recognize  distorted  inputs  as  long  as  the  input  provides  sufficient  information.  Later  in  this 
report  we  will  show  the  characteristics  of  the  associative  memory  by  presenting  distorted 
versions  of  the  stored  images,  e.g.,  rotated,  scaled,  shifted  ones,  etc.  to  the  system  and  see 
how  it  converges.  ,•  ,  ,  .  ■  ••  -  ? 

In  this  report  we  present  holographic  implementations  of  Hopfield’s  model  of  neural 
network  [1,2].  This  model  has  a  very  simple  structure  and  is  easy  to  implement.  However, 
its  principles  and  characteristics,  e.g.,  neuron  thresholding,  global  interconnections  between 
the  neurons,  and  feedback  dynamics,  are  very  flexible  hence  the  results  can  be  easily 
extended  to  other  types  of  neural  networks.  In  the  following  section,  the  basic  principle  of 
the  Hopfield  network  and  its  one-dimensional  implementation  are  reviewed.  The  optical 
implementation  of  the  network  for  processing  two-dimensional  informations  is  presented 
in  Section  3  and  its  experimental  results  are  presented  in  Section  4.  Special  attention 
is  focused  on  the  dynamical  phenomena  of  the  feedback  loop.  Section  5  discusses  the 
trade-off  between  distortion  tolerance  and  image-recognition  capability  of  the  associative 
memory.  Mathematical  modeling  of  this  particular  optical  loop  and  its  convergence  proof 
are  presented  in  Section  6. 

2  The  Hopfield  Model  of  Associative  Memory 

The  basic  structure  of  the  Hopfield  type  neural  network  for  two-dimensional  (2-D) 
informations  is  shown  in  Fig.  1.  It  is  a  single-layer  network  with  feedback.  As  shown  in 
the  figure  there  are  two  ingredients  in  the  network:  the  neurons  and  the  interconnection 
tensor.  The  neurons  are  distributed  in  a  plane  called  the  neural  plane.  The  neurons 
receive  input  images,  perform  nonlinear  thresholding  on  the  received  input,  and  then  re¬ 
emit  output  patterns.  The  output  of  each  neuron  is  connected  to  the  input  of  every  other 
neurons  to  form  a  feedback  network. 

There  are  several  ways  to  implement  a  2-D  neural  plane  [3].  Optoelectronic  tech¬ 
niques  integrate  photo-detectors,  electronic  amplifiers,  and  light  emitting  sources  to  form 
a  neural  plane  [4].  The  thresholding  characteristics  of  the  neurons  are  controlled  by  ad¬ 
justing  the  gains  of  the  electronic  amplifiers.  Photorefractive  BaTiO 3  crystals  operating 
in  a  saturable,  two-beam  amplification  mode  provides  another  alternative  [5]  where  the 
modulation  depth  between  the  two  beams  is  an  adjustable  parameter.  In  our  experiments 
we  use  the  combination  of  an  image  intensifier  and  a  liquid-crystal  light  valve  (LCLV)  to 
simulate  the  neural  plane.  For  the  case  of  2-D  neurons  the  interconnection  pattern  is  a  four- 
dimensional  tensor.  In  our  experiments  the  interconnection  tensor  is  implemented  by  two 
Fourier-transform  holograms.  The  holographic  techniques  of  m along  this  interconnection 
will  be  presented  in  the  next  section.  ^ 

There  are  two  phases  involved  in  operating  the  Hopfield  network,  the  learning  phase 
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Fig.  1:  The  2-D  Hopfield-type  Neural  Network. 


and  the  recalling  phase.  In  the  learning  phase,  the  information  to  be  stored  is  recorded  us¬ 
ing  an  outer  product  scheme.  This  storage  specifies  the  interconnection  strengths  between 
the  neurons.  In  the  recalling  phase,  an  external  input  is  presented  to  the  system.  The 
state  of  the  system  then  evolves  according  to  the  correlation  between  the  input  and  the 
stored  data.  Consider  the  learning  phase  in  the  1-D  case;  we  first  store  M  jV-bit  binary 
words  in  a  matrix  ujti]  according  to 

z".,«r«r<  « '*j\ 

0,  otherwise, 

where  i>,m  =  ±1,  i  =  1 ,  -  -  - ,  JV,  is  the  zth  bit  of  the  mth  memory.  This  matrix  can  be 
calculated  and  recorded  in  a  program  for  simulations  or  it  can  be  plotted  on  a  transparency 
for  optical  experiments.  The  recording  of  this  synaptic  matrix  completes  the  learning 
phase.  In  the  recalling  phase,  external  data  are  fed  into  the  system  and  iterations  are 
induced.  For  example,  suppose  vm0,  the  mth  stored  vector,  is  presented  into  the  system. 
This  vector  is  then  multiplied  by  the  stored  matrix  u>ij,  giving  the  output  of  the  first 
iteration: 


(1) 


t)tm0 


=  sgn 


f  N 

'  M 

sgn  < 

E 

E  »r»r 

U=1 

.m= 1 

(  M 

’  N 

=  sgn  < 

(N  -  l)t>”°  +  y, 

^  m^mO 

_J=3 

where  s  <?«{•}  means  the  thresholding  function, 


(2) 

(3) 


sgn 


if  f(x)  >  0; 
if  f(x)  <  0. 


(4) 


The  thresholded  result  of  the  first  iteration  re-enters  the  system  for  the  next  iteration  so 
that  the  system  evolves  continuously.  We  see  that  in  Eq.  3  two  terms  result  from  each 
iteration:  The  first  term  resembles  the  mth  memory  which  we  call  the  signal;  the  second  is 
the  crosstalk  between  the  mth  vector  and  other  vectors,  which  we  call  the  noise.  We  assume 
that  the  binary  words  are  chosen  randomly  so  that  each  bit  is  statistically  independent;  i.e., 
P[v ,m  =  1]  =  P[ v™  =  —1]  =  1/2;  then  the  signed- to-noise  ratio  (SNR)  before  thresholding 
is 


SNR  = 


E[0,mO] 

■y/varjt)-"0] 

V(N  - \)/(M  -  1) 

fW 

V  M' 


(5) 
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If  N  is  sufficiently  larger  them  M,  then  with  high  probability  each  element  of  £>tm0  equals 
that  of  v™°.  Therefore,  the  stored  vector  vm0  remains  a  stable  state  of  the  system  in 
iterating  operations.  It  was  shown  in  [6,7]  that  if  the  stored  vectors  are  to  be  stable 
states,  then  the  number  of  memories  that  can  be  stored  in  the  system  is  limited  by  M  < 
N/(  4  In  N). 

From  Eq.  2  we  see  that  we  need  three  steps  to  implement  the  Hopfield  network: 
vector-matrix  multiplication,  thresholding,  and  feedback.  This  was  first  done  by  Psaltis 
and  Farhat  using  optoelectronic  techniques  [8,9].  They  used  a  computer-generated  trans¬ 
parency  to  provide  the  interconnection  matrix.  A  1-D  array  of  32  photodiodes  followed 
by  electronic  thresholding  plus  a  1-D  array  of  32  LEDs  was  used  to  simulate  32  neurons. 
The  arrays  were  used  to  detect  and  emit  32-bit  data  vectors.  An  optical  vector-matrix 
multiplier  was  used  to  perform  the  multiplication  between  the  the  data  vector  end  the 
transparency  matrix.  The  multiplied  result  was  detected  by  the  photodiode  array.  The 
detected  signal  of  each  photodiode  was  electronically  thresholded  and  fed  back  to  the  corre¬ 
sponding  LED  for  further  iterations.  This  system  successfully  demonstrated  the  dynamics 
and  the  capability  of  associative  information  recall  of  feedback  neural  networks.  Another 
architecture  using  acousto-optics  also  demonstrated  similar  functions  [10].  In  this  report 
we  are  interested  in  the  processing  of  2-dimensional  images.  The  design  and  its  optical 
implementation  are  presented  in  the  following  section. 

3  Optical  Implementation  of  the  Associative  Memory  Loop 

In  this  section  we  consider  the  implementation  of  the  Hopfield-type  neural  network 
for  2-dimensional  images.  The  interconnection  pattern  for  2-D  images  is  described  by  the 
following  equation: 

M 

w(x,  y;  £,  rj)  =  fm(x,  y),  (6) 

m=  1 

where  /m(x,y)  is  the  mth  image  to  be  stored,  and  M  is  the  total  number  of  images  to 
be  stored.  Note  that  u>(x,y,Z,rj)  is  a  four-dimensional  tensor.  It  cannot  be  implemented 
straightforward  using  a  single  transparency  since  a  2-D  optical  system  has  only  two  spa¬ 
tial  coordinates.  One  could  obtain  additional  variables  by  using  wavelength  multiplexing 
and  time-domain  processing.  In  [8],  Psaltis  and  Farhat  proposed  a  spatial-frequency  mul¬ 
tiplexing  method  in  which  a  2-D  array  of  2-D  holograms,  each  separated  by  different 
spatial-frequencies,  was  used  to  perform  the  4-D  interconnection.  Jang,  et  al.  used  a  2-D 
array  of  N  x  N  diffused  holograms  to  obtain  the  4-D  interconnection  [11,  12]. 

In  this  report  we  approach  this  problem  from  another  point  of  view.  Recall  that  in 
the  recalling  phase,  the  output  of  the  system  is  described  by  the  iterating  equation 

f(x,y,t  +  l)  =  9^J J u(x,y;t,T))f{Z,T),t)d(dT^  ,  (7) 

where  y{-}  represents  the  nonlinear  thresholding  of  the  neurons,  f{x,y,t )  is  the  input  to 
the  system  at  time  t ,  and  f{x,y,t  +  1)  is  the  output  of  the  system.  By  inserting  the 
expression  of  u(x,y\Z,v)  into  this  equation  and  rearranging  the  order  of  integration  and 


summation,  we  obtain 


f(x,y,t  +  1)  =  g 


=  9 

=  9 


fm(x,  J/)/m(£i 


fit,' h*)d£dri 
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{//[.?, 

jj^/m(z,y)  |y J  fm(Lv)f(£,V,i)d£dr)  | 

{£«...)(// 


-x,n-  y,t)d£dri 


r=0,y=o} 


(8) 


From  Eq.  8  we  can  see  that  the  implementation  of  the  2-D  associative  memory  can  be 
achieved  in  three  steps  [13].  The  first  step  is  to  perform  the  inner  product  of  the  input 
image  /  with  each  of  the  memories  fm.  Second,  each  inner  product  is  multiplied  by  the 
associated  memory.  Third,  these  products  are  summed  over  all  memories  and  thresholded 
by  the  neurons  for  iterations.  The  implementation  of  these  steps  matches  holographic 
techniques  exactly.  The  idea  is  shown  in  the  block  diagram  of  Figure  2. 

The  first  step  described  above  can  be  realized  by  sampling  the  correlation  pattern 
at  the  origin  in  Vander  Lugt  system  [14].  The  first  correlator  and  the  pinholes  carry  out 
this  operation.  The  signal  passing  through  the  pinholes  reconstructs  the  images  from  the 
second  correlator.  This  carries  out  the  second  step.  In  the  final  step,  the  reconstructed 
images  are  summed  up  at  the  input  side  of  the  neurons,  in  this  case  the  writing  side  of  the 
LCLV.  The  LCLV  is  used  to  simulate  a  plane  of  neurons  to  perform  the  thresholding  and 
provides  a  signal  for  further  iterations. 

The  operation  of  the  associative  loop  can  be  explained  with  the  aid  of  the  diagram 
shown  in  Fig.  3.  In  this  example  four  images  are  spatially  separated  and  stored  in  the 
Fourier-transform  holograms  Hi  and  H2  as  shown  in  the  figure.  When  the  input  pattern 
A  is  presented  on  the  plane  P\  of  the  system,  the  first  correlator  produces  the  auto¬ 
correlation  pattern  along  with  three  cross-correlations  on  plane  P2 ■  The  pinhole  array  on 
P2  samples  these  correlation  patterns  at  the  center  of  each  pattern  where  the  inner  products 
between  the  input  and  each  of  the  stored  images  form.  Each  of  the  four  beams  that  pass 
through  the  pinholes  act  as  delta  functions,  reconstructing  from  the  second  correlator  the 
four  images  stored  in  hologram  p2-  These  reconstructed  images  are  spatially  translated 
according  to  the  position  of  each  pinhole  and  superimposed  on  plane  Pj .  At  the  center  of 
the  output  plane  of  the  second  correlator  we  obtain  the  superposition  of  the  four  stored 
images.  The  stored  image  that  is  most  similar  to  the  input  pattern  gives  the  strongest 
correlation  signal,  hence  the  brightest  reconstructed  image.  Here  in  Fig.  3  we  show  only 
the  bright  image  that  is  reconstructed  by  the  strongest  auto-correlation  peak.  The  weak 
read-out  signal  that  is  due  to  cross-correlations  can  be  eliminated  somehow  by  thresholding 
by  the  LCLV.  The  output  of  the  LCLV  becomes  the  new  input  image  for  the  loop  and 
forms  a  closed  loop.  The  stable  pattern  that  forms  as  a  recirculating  image  in  the  loop  is 
the  stored  image  that  is  most  similar  to  the  original  input.  This  image  stays  locked  in  the 
loop  even  when  the  external  input  in  turned  off. 

In  the  system  that  we  described  in  the  previous  paragraph,  the  images  are  recorded 
in  a  conventional  Fourier-transform  hologram,  as  shown  in  Fig.  4.  The  four  images,  A, 
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Fig.  2:  Block  Diagram  of  the  Optical  Loop. 
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First  Hj 


B,  C,  D,  in  this  example  are  spatially  separated  at  the  input  plane.  A  single  plane  wave 
is  used  as  the  reference.  We  record  two  identical  holograms,  Hi  and  H 2,  then  put  one  in 
each  of  the  two  correlators  of  Fig.  2.  A  pinhole  array  with  pinhole  separations  equal  to 
that  of  the  corresponding  original  images  is  placed  at  the  correlation  plane  to  sample  the 
correlation  signal. 

The  schematic  diagram  of  the  architecture  of  the  above  design  is  shown  in  Fig.  5;  and 
a  photograph  of  the  experimental  apparatus  is  shown  in  Fig.  6.  In  the  system  shown  in 
Fig.  5,  the  LCLV  at  plane  Pi,  the  beam  splitter  cube  BS 1,  the  lenses  Xj ,  L 2.  and  the 
hologram  Hi  form  the  first  correlator.  The  part  consisting  of  P2 ,  X3,  H 2,  L4,  BS$,  and 
Pi  form  the  second  correlator.  The  input  pattern  is  imaged  onto  the  LCLV  by  lens  L,  and 
through  beam  splitter  BS3 .  A  collimated  argon  laser  beam  illuminates  the  read-out  side  of 
the  LCLV  through  beam  splitters  BS2  and  BS\ .  A  portion  of  the  reflected  light  from  the 
LCLV  that  propagates  straight  through  BS  1,  is  diverted  by  BS2,  and  it  is  imaged  by  lens 
Lo  onto  a  CCD  television  camera.  This  provides  real  time  monitoring  of  the  activity  of  the 
system.  The  portion  of  light  that  is  reflected  by  BSi  into  the  loop  is  Fourier-transformed 
by  lens  L 1  and  illuminates  hologram  Hi.  The  correlation  between  the  input  image  and 
each  of  the  stored  images  is  projected  on  plane  P2.  The  pinhole  array  at  P2  has  spacing 
corresponding  to  the  spatial  separations  of  the  stored  images.  The  remainder  of  the  optical 
system  from  P2  back  to  the  neural  plane  Pi  is  essentially  a  replica  of  the  first  half,  with 
the  hologram  H2  storing  the  same  set  of  images  at  Hi . 

We  now  describe  the  operation  of  this  system  analytically.  The  amplitude  transmit¬ 
tance  of  the  transparency  from  which  the  holograms  are  recorded  is 

M 

f(x,y)  -  ^2  fm(x  ~  am,y -bm),  (9) 

m=  1 


where  fm(x,y)  is  the  amplitude  transmittance  of  the  mth  image  at  the  input  plane  in 
Fig.  5,  ( am ,  bm)  is  the  center  of  fm(x,y)  on  the  (x,y)  plane,  and  M  is  the  total  number 
of  recorded  images.  In  our  experiments,  the  separations  between  different  images  are  the 
same,  i.e.,  am  =  bm  =  ma,  where  a  is  a  constant.  The  M  images  in  Eq.  9  are  Fourier- 
transformed  to  interfere  with  a  reference  wave  e~j2nub  and  recorded  on  a  holographic  plate. 
The  amplitude  transmittance  of  the  developed  hologram  is 


M 


^Fm(u,v)e"'2,r(,10"’+,,6m)  +  e~*2irub 


m 

M 


T(u,v)  = 

=  F^(u,v)e^2‘K^u('am~b^+vbm^  +  Complex  Conjugate  +  DC  terms,  (10) 


where  ( u,v )  is  the  coordinate  in  the  Fourier  plane.  This  hologram  is  placed  in  planes  Hi 
rnd  H2  of  the  system.  In  the  above  equation  we  are  interested  only  in  the  first  term  since 
it  is  the  part  which  the  input  image  will  correlate  with.  Suppose  the  amplitude  of  the 
input  image  at  time  t  is  f(x,y,t)  at  plane  Pi,  then  after  hologram  Hi  the  light  amplitude 
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Fig.  5:  Schematic  Diagram  of  the  Optical  Loop. 
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Fig.  6:  Photograph  of  the  Optical  Loop. 
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diffracted  by  the  first  term  is 


M 


]T  F(U,  V ,  i)F^ («,  t,)ci2ir[«(am-»)+»im] 


(11) 


At  the  correlation  plane,  P2,  the  light  amplitude  is  the  Fourier  transform  of  the  above 
term: 

M 

^  *  S(x'  -  (am  -  b),y'  -  bm) 

m=  1 
M 

=  y  hm(-x',  - y',t )  *  S( X  -  (am  -  b),y'  -  bm ),  (12) 

m 

where  hm(—x',—y',t)  represents  the  correlation  between  /  and  the  memory  /m,  ★  means 
correlation  and  *  means  convolution,  and  (xt,yf)  is  the  coordinate  in  correlation  plane. 
The  above  equation  is  sampled  by  the  array  of  pinholes  having  diameter  W  and  locating 
at  the  positions  ( am  —  b,bm).  In  the  limit  W  —*  0,  the  pinholes  can  be  described  as  delta 
functions.  Thus,  the  signal  passing  through  the  pinholes  becomes 


M 
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m= 1 


hm(-x',-y',t)  *  6(x'  -  (am  -  b),y'  -  bm) 


red 


\  W 

y,  hrn(0,0,t)6(x'  -  (am  -  b),y'  -  bm),  as  W  -►  0. 


(^44^ 


M 


(13) 


m=l 


This  signal  reconstructs  images  from  the  hologram  if 2,  giving 


M  M 

m=  I  m'  =  1 
M  M 

=  yy  f»m(0,0,t)F*.(ii,t;)e->2,rW«m-«mO+*'(6m-6m»)l.  (14) 

m=l  m'  =  l 

This  signal  is  Fourier-transformed  back  to  the  input  side  of  the  neural  plane  Pi .  The  total 
field  amplitude  is 


M  M 


f(x,y,t)  =  EE*  m(0,  0,  t)fm,{x  "I"  (<Zm  Ofn  1)1  y  4*  ( bm  5m/))- 


(15) 


>7i  =  ]  m/= 1 


There  are  M  x  M  images  reconstructed  and  imaged  on  the  neural  plane.  Only  the  terms 
with  m  =  m'  are  on-axis  and  aligned  with  the  original  input  f(x,  y,  t ).  If  we  put  a  window 
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centered  at  the  optical  axis,  with  the  size  equal  to  the  size  of  each  memory,  we  observe 
only  the  terms  where  m  =  m': 

M 

f(x,y,t)  =  hm(0,0 ,t)f„(x,y).  (16) 

m= 1 

The  intensity  of  this  Ught  amplitude  is  detected  by  the  photoconductor  of  the  LCLV  and 
gives  an  output  light  amplitude,  to  the  first  order,  proportional  to  the  detected  intensity. 
Thus,  the  images  are  fed  back  into  the  loop.  The  images  reconstructed  by  the  auto¬ 
correlation  peak  become  stronger  and  stronger  until  the  LCLV  saturates,  whereas  the 
images  reconstructed  by  the  cross-correlations  become  weaker  and  weaker  until  they  die 
out. 

The  key  elements  in  this  optical  loop  are  the  holograms,  the  pinhole  array,  and  the 
threshold  device.  The  holograms  in  this  system  are  thermoplastic  plates.  They  have  a 
resolution  of  800  lines  per  millimeter  over  an  area  of  one  square  inch.  If  we  put  a  mirror 
[13]  or  a  phase  conjugate  mirror  [15,  16,  17]  at  the  pinhole  plane  P2  to  reflect  the  correlation 
signal  back  through  the  system,  then  we  only  need  one  hologram  to  form  the  associative 
memory.  But  then  it  losses  feedback  dynamics.  The  use  of  two  holograms,  however,  not 
only  provides  dynamics  but  also  improves  system  performance.  We  make  the  hologram  at 
H\  with  a  high-pass  characteristic  so  that  the  input  section  of  the  loop  has  high  spectral 
discrimination.  On  the  other  hand,  we  want  the  feedback  images  to  have  high  fidelity  with 
respect  to  the  original  images.  Thus,  the  hologram  at  plane  H2  must  have  broad-band 
characteristics.  We  use  a  diffuser  to  achieve  this  when  making  H2.  Fig.  7(a)  shows  the 
four  original  images.  Fig.  7(b)  shows  the  images  reconstructed  from  the  first  hologram  Hi, 
and  Fig.  7(c)  shows  the  images  reconstructed  from  the  second  hologram  H2.  As  expected, 
Fig.  7(b)  is  a  high-pass  version  of  the  original  image  while  Fig.  7(c)  is  broad  band. 

The  pinhole  array  at  plane  P2  samples  the  correlation  signal  between  the  image  coming 
from  the  LCLV  and  the  images  stored  in  hologram  Hi .  The  pinhole  diameter  used  in  these 
experiments  range  from  45  /zm  to  700  ^m,  depending  on  the  images  to  be  stored  and  the 
desired  system  performance.  If  the  pinhole  size  is  too  small,  then  the  light  that  passes 
through  it  to  reconstruct  the  feedback  image  is  too  weak  to  be  detected  by  the  LCLV  and 
no  iterations  can  occur.  On  the  other  hand,  large  pinholes  introduce  excessive  blurring 
and  cross-talk  in  the  feedback  and  make  the  reconstructed  images  unrecognizable.  The 
pinhole  size  also  affects  the  shift  invariance  property  of  the  loop.  In  order  to  be  recognized, 
the  auto-correlation  peak  from  an  external  image  should  stay  within  the  pinhole.  Larger 
pinholes  allow  more  shift  in  the  input  image.  The  system  performance  under  different 
selections  of  pinhole  diameters  will  be  discussed  in  the  next  section. 

The  purpose  of  the  threshold  device  in  this  system  is  three-fold.  The  first  is  to  bring 
into  the  system  a  coherent  image  from  the  external  input  (although  either  coherent  or 
incoherent  images  can  be  used  as  the  input).  The  second  is  to  provide  a  thresholding 
operation  to  the  feedback  signal  so  that  cross-correlation  is  reduced  in  successive  iterations. 
The  third  is  to  provide  gain  to  the  feedback  signal.  The  optical  signal  is  attenuated  in  the 
loop  because  of  the  diffraction  efficiencies  of  the  Fourier  transform  holograms  and  the  losses 
from  pinholes  as  well  as  lenses  and  beam  splitters.  Therefore,  we  need  to  have  optical  gain 
to  compensate  this  loss.  In  our  system  this  is  achieved  by  adding  an  image  intensifier  at  the 
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Fig.  7:  Images  Stored  in  the  Holograms,  (a)  The  original  images,  (b)  Image 
structed  from  H\.  (c)  Images  reconstructed  from 
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photoconductor  side  of  the  LCLV.  As  described  in  Chapter  2,  the  microchannel  plate  of  the 
image  intensifier  is  sensitive  to  a  minimum  incident  intensity  of  approximately  1  nW/cm 2 
and  reproduces  the  input  with  an  intensity  104  times  brighter  (10  yW/cm 2).  This  is  bright 
enough  to  drive  the  LCLV.  If  we  use  a  beam  with  intensity  equal  to  10  mW/cm2  to  read 
the  LCLV,  then  the  intensity  of  the  output  light  is  approximately  1  mW/cm2.  Thus,  the 
combination  of  the  image  intensifier  and  the  LCLV  provides  optical  gain  up  to  106.  This 
optical  gain  is  similar  to  a  sigmoid  function  and  its  slope  can  be  adjusted  by  changing 
the  bias  voltage  of  the  image  intensifier.  In  Section  5  we  will  see  that  the  setting  of  the 
gain  is  the  key  parameter  that  mediates  the  trade-off  between  distorsion  invariance  and 
the  discrimination  capability  of  the  loop. 

4  Experimental  Results 

In  this  section  we  show,  experimentally,  many  interesting  properties  of  the  optical 
associative  loop.  These  include  retrieval  of  complete  images  from  partied  inputs,  recognition 
of  the  shifted,  rotated  and  scaled  images,  error-correction  capability,  and  neural  network 
dynamics.  The  most  interesting  among  these  is  the  system  dynamics.  This  is  a  unique 
property  of  a  neural  network  with  feedback  and  threshold.  The  state  of  the  system  evolves 
in  time  and  the  dynamics  are  determined  by  the  operation  parameters  .  The  dynamics 
provide  a  trade-off  for  optimal  system  design.  The  dynamics  of  the  associative  memory  is 
presented  in  the  following  section  and  the  trade-off  issue  will  be  discussed  in  Section  5. 

4.1  The  Dynamics  of  the  Associative  Loop 

According  to  the  characteristics  of  the  optical  components,  i.e.,  active  or  passive, 
the  optical  associative  loop  of  Fig.  5  can  be  lumped  into  a  simplified  diagram  as  shown 
in  Fig.  8(a).  Since  the  LCLV  provides  optical  gain  and  thresholding  to  the  signal,  it  is 
represented  as  the  component  Gain  in  Fig.  8(a).  The  other  parts  of  the  loop  are  all  lossy 
components  and  are  represented  as  the  component  Loss  in  Fig.  8(a).  This  includes  the 
beam-splitter  cubes,  the  lenses,  the  mirrors,  the  pinhole  array,  and  the  holograms.  The 
circulation  of  the  optical  signal  in  the  loop  is  also  shown  by  the  arrows  in  the  figure.  Note 
that  in  the  loop  the  output  of  the  neurons  forms  the  input  to  the  lossy  part,  and  the  output 
of  the  lossy  part  becomes  the  new  input  for  the  neurons  in  the  next  iteration. 

Let  us  consider  the  loop  dynamics.  The  dynamics  of  the  recall  process  can  be  described 
by  using  an  iteration  map  formed  by  the  gain  and  loss  curves  as  shown  in  Fig.  8(b).  In 
the  figure  the  sigmoid  curve  represents  the  input-output  response  of  the  neurons.  The 
slope  of  the  curve  is  proportioned  to  gain  of  the  neurons,  whereas  the  slope  of  the  straight 
line  is  proportional  to  loop  loss  because  of  the  holograms  and  pinholes  and  is  drawn  on 
the  same  diagram  as  the  input-output  response  of  the  neurons.  The  intersection  point  of 
this  line  with  the  neural  gain  curve  at  Q\  determines  the  loop  threshold  level,  and  the 
intersection  point  Q2  represents  a  stable  point.  If  the  initial  condition  of  the  neuron  is 
above  the  threshold  level  8\ ,  such  as  Jj  shown  in  the  figure,  the  signal  grows  after  each 
iteration  until  it  arrives  and  latches  at  Q2.  On  the  other  hand,  if  the  initial  condition  is 
below  B\ ,  such  as  /2  shown  in  the  figure,  the  signal  will  decay  to  zero.  The  number  of 
iterations  depends  on  the  distance  of  the  initial  condition  from  the  threshold. 

The  loop  dynamics  was  measured  by  controlling  two  shutters  as  shown  in  Fig.  8a. 
The  temporal  response  of  the  loop  to  an  input  pattern  is  shown  in  Fig.  9.  The  lower  trace 
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Fig.  8:  Iterations  in  the  Optical  Loop,  (a)  The  gain  and  loss  components  in  the  loop, 
(b)  The  iterations  in  the  loop. 
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represents  the  intensity  of  the  external  input  image  and  the  upper  trace  represents  the 
corresponding  light  intensity  detected  at  the  loop  output.  Before  time  <1,  both  shutters 
are  OFF  and  the  responses  are  low.  At  time  t1  the  input  shutter  is  turned  ON;  hence  the 
lower  trace  becomes  high,  but  the  loop  shutter  remains  OFF.  The  upper  trace  shows  the 
corresponding  response  of  the  neurons  to  the  external  input.  The  rise  time  of  the  LCLV  is 
approximately  one  second  in  this  experiment.  At  time  <2  the  loop  shutter  is  turned  ON 
and  the  loop  is  closed.  The  feedback  signal  arrives  at  the  neurons  as  an  additional  input 
and  iteration  occurs.  From  Fig.  9(a)  we  see  that  it  takes  about  two  seconds  for  the  loop 
to  reach  a  stable  state.  At  time  t$  the  input  shutter  is  turned  OFF;  hence  the  lower  trace 
becomes  low.  However,  the  loop  remains  latched  to  a  stable  state,  which  is  one  of  the 
stored  images.  Fig.  9(b)  shows  the  same  experiment  but  with  input  intensity  reduced  to 
one-third  of  the  first  input.  The  first  rise  of  the  upper  trace  shows  that  the  rise  time  of  the 
neurons  remains  one  second.  And  the  second  rise  of  the  upper  trace  shows  that  it  takes 
approximately  four  seconds  for  the  loop  to  reach  its  stable  state.  However,  after  the  input 
is  turned  off,  the  loop  gives  the  same  output  intensity.  This  example  shows  that  initial 
conditions  affect  the  dynamics  of  the  loop  but  do  not  affect  the  final  state  of  the  system. 
Fig.  10  shows  the  iteration  map  of  this  experiment  where  the  initial  input  J2  is  lower  them 
I\.  It  shows  that  J2  takes  more  iterations  to  reach  the  stable  state,  but  the  final  state  is 
the  same  as  that  of  Ji . 

Since  the  external  input  does  not  affect  the  shape  of  the  final  state,  but  rather  selects 
which  state  is  produced,  we  can  build  a  degree  of  invariance  in  the  system  since  a  shifted, 
rotated  or  scaled  version  of  a  stored  image  can  recall  the  stored  image.  The  effect  of  such 
distortions  of  the  input  image  is  to  decrease  the  level  of  the  initial  condition.  As  long 
as  the  initial  condition  is  above  the  loop  threshold  (0  in  Fig.  8(b)),  the  loop  is  always 
brought  to  the  stable  state  that  is  most  similar  to  the  initial  input.  This  means  that  the 
dynamics  is  determined  by  the  relative  position  of  the  initial  input  with  respect  to  the  loop 
threshold.  The  initial  condition  is  determined  by  the  degree  of  distortion  of  the  external 
image  as  well  as  its  light  intensity.  On  the  other  hand,  the  loop  threshold  is  determined 
by  system  parameters  such  as  the  neural  gain  and  loop  loss.  In  the  next  section  we  give 
experimental  results  of  system  invariances  against  various  distortions.  The  problem  of  how 
to  select  optimal  system  parameters  to  make  the  trade-off  between  distortion  tolerance  and 
capability  of  recognizing  correct  images  will  be  presented  in  Section  5. 

4.2  Retrieval  of  the  Complete  Image  from  a  Distorted  Input 

In  this  section  we  show  experimental  results  of  the  invariance  property  of  the  memory 
loop  in  recognizing  a  distorted  image.  The  images  stored  in  the  loop  are  the  four  faces 
shown  in  Fig.  7(a).  They  are  recorded  as  a  Fourier- transform  hologram  as  described  in 
Section  3. 

Fig.  11(a)  shows  the  response  of  the  memory  when  the  half  face  of  a  recorded  image 
is  presented  to  the  system  with  the  loop  shutter  OFF.  This  sets  the  initial  condition  of 
the  loop.  We  then  turn  the  loop  shutter  ON  to  close  the  feedback  loop.  The  signal  then 
circulates  in  the  loop  and  the  state  evolves.  After  many  iterations  the  loop  reaches  the 
stable  state  and  the  complete  face  appears.  The  time  for  this  process  ranges  from  less 
than  one  second  to  several  seconds,  depending  on  the  initial  conditions  and  the  system 
parameters.  The  complete  image  is  locked  in  the  loop  even  after  the  external  input  is 
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Fig.  9:  Temporal  Response  of  the  Loop,  (a)  Strong  input,  (b)  Weak  input. 
Timing:  <1  =  Input  ON,  t2  =  Feedback  ON,  t3  =  Input  OFF. 


18 


turned  OFF.  Fig.  11(b)  shows  the  system  output  at  the  moment  the  loop  shutter  is  ON. 
We  see  that  the  feedback  image  is  superimposed  on  the  external  input.  Fig.  11(c)  shows 
the  output  0.4  seconds  after  the  feedback  loop  is  closed,  and  Fig.  11(d)  shows  the  output 
0.8  seconds  after  the  feedback  loop  is  closed.  Fig.  11(e)  shows  the  complete  image  recalled 
from  the  loop  after  2  seconds.  Fig.  11(f)  shows  that  after  we  remove  the  external  input 
the  recalled  image  is  latched  in  the  loop.  Fig.  12  shows  the  temporal  sequence  of  the 
same  experiment  but  with  smother  half- face  as  the  input.  This  experiment  shows  that 
the  external  input  to  the  associative  memory  does  not  need  to  be  exactly  the  same  as  the 
recorded  images. 

Now  we  use  a  rotated  version  of  one  of  the  stored  images  as  the  input  to  investi¬ 
gate  the  rotation  invariance  capability  of  the  loop.  Fig.  13(a)  shows  the  response  of  the 
memory  when  a  rotated  version  of  a  recorded  image  is  presented  into  the  system  with  the 
loop  shutter  OFF.  The  input  is  rotated  by  6  degrees  with  respect  to  the  original  image. 
Fig.  13(b)  shows  the  memory  output  at  the  moment  the  feedback  loop  is  closed.  The  loop 
state  then  evolves  to  give  the  original  image.  The  temporal  sequence  of  this  evolving  is 
shown  in  Fig.  13(c)  to  Fig.  13(e).  In  this  experiment  the  degree  of  rotation  of  the  input 
image  sets  the  initial  condition  of  the  loop.  More  rotation  means  more  distortion;  hence 
the  initial  condition  is  farther  away  from  the  stored  memory.  Thus,  the  loop  needs  more 
iterations  to  arrive  at  the  stable  state.  Fig.  14(a)  shows  this  result. 

In  the  figure  the  upper  curve  represents  the  stable  state  intensity  of  the  output  image 
and  the  lower  curve  represents  the  time  for  the  loop  to  reach  stable  states,  both  as  a 
function  of  rotation  angle.  It  shows  that  as  the  input  is  rotated,  it  takes  a  longer  time  to 
be  recognized.  However,  once  the  loop  converges  to  the  stable  state,  the  output  intensity 
is  always  the  same  regardless  of  initial  rotation.  The  figure  also  shows  that  the  output 
intensity  drops  to  zero  when  the  initial  input  is  rotated  over  8  degrees.  This  means  that 
the  initial  condition  is  below  the  loop  threshold  and  the  rotated  image  is  not  recognizable. 
One  way  to  increase  the  tolerance  of  rotation  is  to  increase  the  neural  gain  so  that  it  can 
detect  weaker  feedback  signals  from  the  distorted  input.  Fig.  14(b)  shows  the  result  of 
the  same  experiment  but  with  the  neural  gain  10  times  higher  compared  to  that  used  in 
Fig.  14(a).  The  rotation  tolerance  now  increases  to  16  degrees.  One  might  infer  that  we 
can  obtain  more  tolerance  simply  by  increasing  the  neural  gain.  This  is  not  true  since  high 
gain  also  enhances  crosstalk  and  the  loop  may  not  converge  to  the  correct  image.  This 
problem  will  be  addressed  in  Section  5. 

The  third  experiment  on  distortion  tolerance  is  scale  invariance.  Fig.  15(a)  shows  the 
response  of  the  memory  when  the  scaled  version  of  the  face  of  a  recorded  image  is  presented 
to  the  system  with  the  loop  shutter  OFF. 

This  face  is  85%  of  the  size  of  the  original  stored  image.  After  we  turn  ON  the 
feedback  shutter,  the  loop  evolves  to  the  stable  state.  The  temporal  sequence  of  this 
evolving  behavior  is  shown  in  Fig.  15(b)  to  Fig.  15(e).  We  now  remove  the  external  input 
by  turning  OFF  the  input  shutter.  Fig.  15(f)  shows  that  the  loop  remains  latched  to 
the  original  image.  The  convergence  time  in  this  case  is  1.5  seconds.  This  is  longer  than 
what  would  be  required  if  the  input  were  the  original  image.  However,  the  two  cases  give 
the  same  output  intensity.  When  the  input  image  is  further  scaled  down  to  70%  of  the 
original  size  we  need  to  increase  the  neural  gain  from  104  to  105  for  the  loop  to  recognize 
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Fig.  11:  Retrieval  of  the  complete  Image  from  the  Partial  Input,  (a)  The  partial  input 
at  t  =  0.  (b)  t  =  0+  (Loop  closed),  (c)  t  =  400  ms.  (d)  t  =  800  ms.  (e)  t  =  2  sec.  (f) 
Input  OFF. 


(e)  (f) 


Fig.  12:  Retrieval  of  the  complete  Image  from  the  Partial  Input,  (a)  The  partial  input 
at  t  =  0.  (b)  t  =  0+  (Loop  closed),  (c)  t  =  480  ms.  (d)  t  =  800  ms.  (e)  t  =  2  sec.  (f) 
Input  OFF. 
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Fig.  13:  Retrieval  of  the  complete  Image  from  the  Rotated  Input,  (a)  The  input  at 
t  -  0.  (b  )t  =  0+  (Loop  closed),  (c)  t  =  1.8  sec.  (d)  t  =  3.6  sec.  (e)  t  =  4.8  sec.  (f)  Input 

OFF. 
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Fig.  14:  Rotation  Invariance  Property  of  the  Optical  Loop,  (a)  Optical  gain  =  104. 
(b)  Optical  gian  =  105.  (o:  Output  intensity,  o:  Loop  rise  time.) 
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Fig.  15:  Retrieval  of  the  Complete  Image  from  the  Seeded  Input,  (a)  Th*:  input  at 
t  =  0.  Image  size  =  85%.  (b)  t  =  0+  (Loop  closed),  (c)  t  =  1.8  sec.  (d)  t  =  3.0  sec.  (e) 
t  =  4.8  sec.  (f)  Input  OFF. 
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the  image.  But  this  high  gain  results  in  low  discrimination  such  that  an  input  image  that 
is  not  stored  is  also  incorrectly  recognized.  These  results  are  consistent  with  the  dynamics 
and  the  invariance  properties  that  we  discussed  above. 

In  principle,  this  loop  is  shift  invariant  since  the  images  are  stored  in  the  Fourier- 
transform  holograms.  If  one  of  the  stored  images  appears  at  the  input  plane,  there  will 
be  a  bright  spot  at  the  correlation  plane.  If  the  input  image  shifts,  the  correlation  peak 
will  also  shift  to  a  corresponding  position.  However,  if  the  peak  shifts  out  of  the  pinhole 
position,  then  we  lose  the  feedback  signal  and  the  image  cannot  be  latched  in  the  loop. 
Only  when  the  shift  equals  an  exact  image  spacing  can  the  correlation  peak  pass  through 
the  pinhole  to  close  the  feedback  path.  But  then  the  image  that  appeared  in  the  window 
would  be  different  from  the  previous  one.  Hence,  this  loop  is  shift-invariant  only  within 
the  pinhole  size.  Without  the  pinholes  the  cross- correlation  noise  and  the  auto-correlation 
peak  would  be  fed  back  to  the  loop  together  and  the  reconstructed  images  would  not  be 
recognizable.  There  is  a  compromise  between  pinhole  size  and  loop  performance.  Small 
pinholes  allow  good  memory  discrimination  and  sharp  reconstructed  images,  but  can  also 
cut  the  signal  to  below  the  level  that  can  be  detected  by  the  threshold  device  and  reduce 
the  shift  tolerance  of  the  system.  The  function  of  the  pinhole  array  in  this  system  might 
also  be  met  by  using  a  nonlinear  spatial  light  modulator,  in  which  case  we  can  achieve  full 
shift  invariance  [18,  19]. 

Fig.  16(a)  shows  the  read-out  image  from  the  LCLV,  which  comes  from  an  external 
input  shifted  away  from  its  stored  position.  This  shift  moves  its  correlation  peak  so  that 
it  does  not  match  the  position  of  the  pinhole.  Thus,  there  is  no  feedback  signal  going 
through  the  loop.  If  we  cut  off  the  input  image,  the  read-out  image  will  die  out  with 
a  characteristic  time  of  about  50  to  400  ms,  corresponding  to  the  response  time  of  the 
LCLV.  Now  we  shift  the  input  image  around,  trying  to  search  for  the  correct  position. 
Once  the  input  image  comes  close  enough  to  the  correct  position,  the  correlation  peak 
passes  through  the  right  pinhole,  giving  a  strong  feedback  signal  superimposed  with  the 
external  input  on  the  neurons.  The  total  signed  then  goes  through  the  feedback  loop  and 
is  amplified  continuously  until  the  neurons  are  saturated.  Fig.s  16(b)  to  (e)  show  the 
temporal  sequence  of  this  development  from  the  moment  that  a  sufficient  portion  of  the 
correlation  peak  passes  through  the  pinhole  to  the  complete  recall  of  the  original  image. 
Fig.  16(f)  shows  the  image  in  the  loop  with  the  input  turned  OFF  and  the  memory  latched. 
In  this  experiment  the  pinhole  diameters  are  90  / im .  As  long  as  the  correlation  peak  is 
close  enough  to  the  pinhole,  the  original  image  can  be  recalled.  Depending  on  how  far  the 
input  image  is  shifted  from  the  original  position  and  the  optical  gain  of  the  neurons,  the 
time  required  for  the  loop  to  reach  a  stable  state  is  between  two  to  several  seconds.  The 
lower  curve  of  Fig.  17(a)  shows  the  loop  rise  time  as  a  function  of  shift  when  the  neural 
gain  is  104. 

The  upper  curve  shows  the  output  intensity  as  a  function  of  shift.  It  shows  that  the 
shift  tolerance  for  this  particular  image  is  220  fim  and  that  the  loop  converges  to  the  image 
with  the  same  intensity.  If  the  shift  is  more  than  220  /im,  the  feedback  signal  is  too  weak 
to  trigger  the  loop  and  the  output  is  zero.  If  we  increase  the  neural  gain  by  a  factor  of  ten, 
then  the  shift  invariance  increases  to  about  500  /im.  This  is  shown  in  Fig.  17(b).  However, 
sidelobes  are  also  amplified  in  the  loop;  hence  the  image  may  be  uncorrectly  recognized. 
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Fig.  16:  Retrieval  of  the  Complete  Image  from  the  Shifted  Input,  (a)  The  input  at 
t  =  0.  (b )t~  0+  (Loop  closed),  (c)  t  =  2.4  sec.  (d)  t  =  3.0  sec.  (e)  t  =  0.4  sec  after  the 
input  is  OFF.  (f)  Stable  state. 


Fig.  17:  Shift  Invariance  Property  of  the  Optical  Loop,  (a)  Optical  gain  =  104.  (b) 
Optical  gian  =  105.  (o:  Output  intensity,  o:  Loop  rise  time.) 
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The  dynamic  and  invariance  properties  of  the  associative  loop  shown  above  imply  that 
the  associative  memory  has  error-correcting  capability,  since  the  input  does  not  have  to  be 
the  same  as  the  original  memory  to  be  recognizable.  We  show  this  capability  particularly 
by  using  four  words  as  memory.  Fig.  18(a)  shows  the  four  words  recorded  in  the  Fourier- 
transform  holograms  of  our  system.  Fig.  18(b)  shows  an  input  word  that  has  three  spelling 
errors.  However,  it  has  four  correct  letters,  O,  P,  I,  A,  at  the  correct  positions.  Hence 
the  correlation  of  (OP4I&A)  with  the  memory  gives  sufficient  feedback  signal  to  trigger 
the  loop  iterations.  Fig.  18(c)  shows  the  word  recalled  from  the  loop,  superimposed  on 
the  input.  Fig.  18(d)  shows  the  loop  latched  to  the  correct  word  after  the  input  is  turned 
OFF.  If  we  increase  the  neural  gain,  the  loop  would  be  able  to  recognize  more  erroneous 
words.  But  then  it  will  lose  the  discrimination  capability  because  a  completely  different 
word  would  also  induce  enough  feedback  for  the  word  to  be  recognized  as  one  of  the  stored 
words. 

5  Trade-Off  Between  Distortion  Tolerance  and  Discrimination  Capability 

The  experimental  results  shown  in  the  above  subsection  demonstrates  the  distortion- 
invariance  capability  of  the  associative  loop.  The  input  images  do  not  have  to  match 
the  memory  exactly.  Furthermore,  by  raising  the  neural  gain,  no  matter  how  much  we 
change  the  initial  condition  by  rotating,  shifting,  and  scaling  the  input  image,  the  loop 
can  always  be  made  to  produce  an  image  as  a  stable  state.  But  the  ability  to  correctly 
recognize  a  stored  image  from  a  distorted  input  and  the  discrimination  capability,  i.e.,  the 
ability  to  distinguish  images  from  one  another,  compromise  each  other.  If  there  is  too 
much  gain,  then  just  shining  a  flashlight  at  the  input  of  the  system  causes  it  to  lock  on 
to  one  of  its  stable  states.  If  the  gain  is  set  too  low,  then  even  an  input  that  is  a  slightly 
distorted  version  of  one  of  the  stored  images  is  not  recognizable.  In  particular,  there  are 
two  parameters  under  our  control  that  can  affect  the  gain  in  the  loop:  The  gain  of  the 
neurons  and  the  size  of  the  pinholes. 

The  importance  of  selecting  an  optimal  neural  gain  can  be  explained  by  the  loop 
iteration  map.  Fig.  19  shows  the  iteration  map  of  a  low  gain  loop.  In  the  figure  there  is  no 
intersection  between  the  gain  curve  and  the  loss  line,  and  the  gain  curve  is  always  below 
the  loss  line.  Therefore,  no  matter  how  well  the  input  image  matches  the  stored  images 
and  no  matter  how  bright  the  correlation  peak  is,  the  output  of  the  neuron  is  always  below 
the  loss  line.  Hence,  the  loop  signal  becomes  weaker  and  weaker  in  successive  iterations 
until  finally  it  decays  to  zero.  Fig.  20(a)  shows  the  response  of  the  loop  to  an  input  image, 
with  the  feedback  shutter  OFF.  Fig.  20(b)  and  (c)  shows  the  output  after  the  loop  is 
closed.  Fig.  20(d)  and  (e)  shows  the  loop  output  after  the  input  image  is  OFF.  The  loop 
image  decays  because  the  neural  gain  is  too  low. 

On  the  other  hand,  too  much  neural  gain  also  causes  problem.  Fig.  21  shows  the 
iteration  map  with  high  gain.  In  this  figure  the  gain  curve  is  higher  than  the  loss  line 
and  they  intersect  at  a  low  value.  Hence,  the  loop  has  a  low  threshold.  Therefore,  a  small 
correlation  signal  is  enough  to  trigger  the  loop  to  evolve.  As  shown  in  the  above  subsection, 
this  loop  can  tolerate  more  image  distortions  such  as  rotation,  scaling,  shift,  etc.,  and  can 
still  recognizethem.  This  means  that  the  loop  has  a  very  large  radius  of  attraction  for  each 
memory  state.  However,  it  also  means  that  its  discrimination  capability  to  distinguish 
different  images  is  poor.  It  is  quite  possible  that  the  loop  recognizes  other  images  as  one  of 
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Fig.  18:  Error- Correction  Capability  of  the  Loop,  (a)  The  stored  images,  (b)  External 
input  with  errors,  (c)  Feedback  image  superimposes  with  the  input  image,  (d)  The  stable 
state  of  the  loop  after  the  external  is  OFF . 
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Fig.  19:  Iteration  Map  of  Low  Optical  Gain. 
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Fig.  20:  Loop  dynamics  of  Low  Optical  Gain,  (a)  The  input  at  t  =  0.  (b)  t  =  0+ 
(Loop  closed),  (c)  t  =  3  sec.  (d)  @  Input  OFF.  (e)  t  =  1.2  sec  after  the  input  is  OFF.  (f) 
t  =  1.8  sec  after  the  input  is  OFF. 


the  stored  images.  Fig.  22(a)  shows  an  image  that  is  not  stored  in  the  memory.  Fig.  22(b) 
to  (d)  show  that  the  unfamiliar  image  triggers  the  loop  to  evolve  after  the  feedback  shutter 
is  turned  ON.  Fig.  22(e)  shows  that  after  the  input  image  is  turned  OFF,  the  loop  is 
locked  to  one  of  the  stored  images.  This  shows  the  loop  making  an  incorrect  recognition. 

The  gain  required  to  sustain  the  iteration  loop  is  determined  by  the  loop  loss.  The 
factors  in  this  system  that  decide  the  loss  axe  the  pinhole  size,  the  hologram  diffraction 
efficiency,  and  the  reflections  from  optical  components.  Among  them  the  pinhole  size  is  the 
key  factor  since  the  other  components  are  generally  fixed.  We  use  Fig.  3  as  an  example. 
Let  fi(x,  y ),  i  =  1,  2, 3,4,  represent  the  images  of  the  letters  A,  B,  C,  D,  respectively,  and 
let  the  pinhole  size  be  \V .  Then  the  reconstructed  images  in  the  window  at  Pi  can  be 
shown  to  be 

4 

£[M*,tf)rect(£)wct(£)]  *  f,(x,y)  (17) 

»=1 

where  rect(-^r)rect(-^r)  represents  the  finite  size  of  the  pinholes,  *  represents  the  convolu¬ 
tion  operation,  <7n(x,y)  the  auto-correlation  of  A,  and  gu,i  ^  1,  the  cross-correlations  of 
A  with  B,  C,  D,  respectively.  We  see  that  the  images  are  blurred  by  the  finite  dimen¬ 
sion  of  the  pinholes.  Decreasing  W  gives  better  image  quality,  but  we  need  to  increase 
the  gain  of  the  neurons  to  compensate  for  the  loss  caused  by  the  small  pinholes.  At  the 
other  limit,  if  the  pinhole  size  is  increased,  we  do  not  need  very  high-gain  neurons  but  the 
image  quality  deteriorates.  Fig.  23  shows  the  effect  of  the  pinhole  size  on  the  stable-state 
loop  images.  In  the  limit  where  W  becomes  infinitely  large,  the  reconstructed  image  in 
the  window  at  P\  becomes  a  superposition  of  all  the  stored  images,  each  approximately 
equally  strong,  and  severely  blurred.  Fig.  24  shows  the  temporal  sequence  of  the  loop 
signal  under  the  condition  of  infinite  pinhole  size.  It  is  seen  that  final  image  in  the  loop  is 
totally  unrecognizable. 

Thus,  there  is  an  optimum  pinhole  size  and  an  optimum  neural  gain.  Fig.  25  shows  the 
experimental  results  of  minimum  gain  required  and  maximum  gain  allowable  for  the  loop 
to  sustain  a  stable  memory  as  a  function  of  pinhole  size.  Below  the  minimum  gain  the  loop 
can  not  recognize  any  image  in  the  sense  that  once  the  external  input  is  cut  off,  the  loop 
activity  decays  to  zero.  Above  the  maximum  gain  the  loop  loses  discrimination  capability 
such  that  any  input  image,  even  a  flashlight,  will  trigger  the  loop  into  a  stable  state.  Note 
that  the  minimum  gain  increases  when  the  pinhole  size  is  increased  to  more  than  250  jim. 
This  is  because  the  reconstructed  images  are  blurred  so  much  that  the  correlation  peaks 
are  weakened  and  the  losses  in  the  loop  are  increased.  Fig.  25  shows  that  the  optimum 
pinhole  size  in  this  system  is  in  the  range  of  70  /. im  to  150  \im.  We  choose  90  /im  for  most 
of  the  experiments.  Although  this  measurement  is  particular  for  the  images  we  used,  the 
above  behavior  is  true  in  general. 

6  Neural  Network  Model  for  the  Memory  Loop 

The  architecture  of  the  optical  associative  memory  that  we  presented  in  the  preceding 
sections  is  a  Hopfield-type  network.  In  the  system,  the  neurons  are  simulated  by  the  LCLV, 
and  the  interconnections  between  the  neurons  are  achieved  by  holographic  gratings.  In  the 
following,  we  present  an  analysis  of  the  neural  network  model  of  the  optical  associative- 
memory  loop.  Based  on  the  model,  we  will  discuss  the  convergence  properties  of  the  system 
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Fig.  21:  Iteration  Map  of  High  Optical  Gain. 
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Fig.  23:  Output  Image  for  Different  Pinhole  Sizes,  (a)  40  ^m.  (b)  90  n m .  (c)  180  //m 
(d)  400  fim. 
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Fig.  24:  Loop  Behavior  without  the  Pinholes. 
(Loop  closed),  (c)  t  =  0.6  sec.  (d)  t  =  2  sec 
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Fig.  25:  Optimum  Pinhole  Size  and  Optical  Gain. 


and  the  stability  of  the  stored  memories. 

Recall  from  Eq.  16  that  the  feedback  signal  of  each  iteration  in  the  optical  loop  can 
be  written  as 

M 

f°(*,v,t)=£(f*/m)(0,0,t)r’(*,v),  (18) 

m=l 

where  fm(x,y)  (  m  =  1...M)  are  the  stored  images,  f(x,y)  is  the  input  image  to  the 
LCLV,  ★  is  the  cross-correlation  operator,  and  /m*  is  the  complex  conjugate  of  /m.  Eq.  18 
shows  that  the  feedback  signal  in  the  loop  is  the  superposition  of  the  reconstructed  images 
from  the  second  hologram,  each  being  weighted  by  the  cross- correlation  of  the  input  with 
the  images  stored  in  the  first  hologram.  In  the  optical  system,  we  make  the  first  hologram 
H i  with  high-pass  characteristics.  The  high-pass  Fourier- transform  hologram  was  obtained 
by  adjusting  the  ratio  of  the  recording  intensities  of  the  reference  beam  and  the  object  beam 
such  that  the  high-frequency  part  has  good  modulation  depth  while  the  low-frequency 
parts  were  overexposed.  Thus,  in  reconstruction  there  is  no  diffraction  from  low-frequency 
gratings,  and  only  high-frequency  components  reconstruct  the  image.  The  reconstructed 
image  contains  only  the  edges  of  the  original  image,  as  was  shown  ;n  Fig.  7(b).  The 
characteristics  of  the  high-pass  hologram  can  be  described  by  subtracting  the  low-frequency 
portion  from  the  original  spectrum,  which  can  be  represented  approximately  by  removing 
the  dc  signed.  Using  the  high-pass  hologram  as  H j  in  the  optical  loop,  the  cross-  correlation 
term  /  *  fm  in  Eq.  18  now  should  be  replaced  by  the  convolution  of  the  input  f(x,y )  with 
the  stored  images  gm(x,y),  i.e.,  /  ★  gm ,  where  g(x,y )  is  the  high-pass  version  of  /(x,y). 
Since  gm(x,  y)  is  the  original  image  with  the  dc  level  removed,  it  can  be  described  by 

sm(*,y)  =  /m(x,y)- (19) 

where  the  integration  is  performed  over  the  finite  size  of  the  images. 

Instead  of  taking  continuous  functions  /m(x,y),  /(x,y),  etc.,  we  will  approximate 
by  sampling  discrete  points  (pixels),  indexed  by  i,  j,  etc.  (The  number  of  the  sampled 
points  N  should  be  greater  or  equal  to  the  SBP  of  the  optical  system.)We  can  approximate 
(/★ym)(0,0)  by  summation  of  the  inner  product 

N  N 

(/*9m)(o,o)  =  £]T/,is;;  (20) 

/=1  k=  1 

and  write 

=  m  -  4  E  /,?-  (2D 

«.;=i 

The  feedback  signal  can  thus  be  written  as 

M  N  N 

A?  =  E  [EE^ie]/"- 

m=l  1=1  k=  1 
N  N 

= y^y 'wnjk  fik,  (22) 

1=1  k=  1 
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where 


=  E 

m=l 

=  E  [/»  -  (4  E  /<?)]/,?•■  <23> 

m=l  l,k=l 

Note  that  the  matrix  W  is  nonsymmetric  and  that  its  diagonal  terms  are  not  zero  in  this 
case. 

For  simplicity  of  discussion,  we  use  one-dimensional  signals  in  the  following  analysis. 
The  extension  to  the  two-dimensional  case  is  straightforward.  We  also  assume  that  the 
signals  are  real.  In  fact,  this  may  not  be  true  in  the  real  system,  because  the  nonunifor¬ 
mity  and  phase  distortions  of  the  optical  components  and  the  LCLV  may  cause  signals  to 
become  complex.  However,  the  analysis  then  becomes  difficult  if  we  try  to  take  this  into 
account.  Furthermore,  the  intensity  of  the  signal  is  detected  in  each  iteration,  and  the 
phase  information  does  not  accumulate  in  the  system;  thus,  our  assumption  is  reasonable. 
Following  these  arguments,  the  Wij  for  the  1-D  case  is  modified  as 

M 

wij  =  ^  ,{xi  ~am)xi  ,  (24) 


where 


°m  ~  A?  S  XT 


is  the  average  level  of  image  m.  The  feedback  signal  shown  in  Eq.  22  can  then  be  written 


Therefore,  the  feedback  signal  after  one  cycle  and  before  thresholded  by  the  neurons  can 
be  obtained  by  inserting  Eq.  24  into  26,  which  gives 


N  M 


yi  =  £(X(xr  -a"*)xr)*; 


j=l  m=l 
M  N 


=  E  [(E*r*>)  -  (E1')0-]1!”- 


m=l  j=  1 


Note  that  in  this  case  the  xj"  and  Xj  are  unipolar,  i.e.,  0  or  1;  whereas  y<  can  be  biploar 
because  is  bipolar.  The  above  expression  can  be  written  in  a  matrix  form, 


y  =  J2  [(x-xm)  -  (I>>)a">]x">- 
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The  signal  y  is  fed  back  at  the  neuron  plane  and  is  thresholded  by  the  neurons  to  give  the 
signal  for  the  next  iteration.  Thus,  the  new  signal  for  the  next  iteration  is 

(29) 

This  process  then  repeats  until  the  loop  reaches  an  equilibrium  state.  Clearly,  if  we  want 
to  investigate  the  equilibrium  states  and  the  stability  characteristics,  we  should  solve  the 
dynamic  equations  of  the  system.  However,  before  we  go  to  that  step,  we  can  qualitatively 
predict  the  performance  of  the  loop  simply  by  inspecting  the  physical  meaning  of  each 
term  of  the  feedback  signal. 

As  is  seen  in  Eq.  28,  the  total  feedback  signed  is  the  weighted  sum  of  the  stored 
images  xm.  The  weight  of  each  xm  is  determined  by  the  two  terms  in  the  square  bracket. 
The  term  x  •  xm  is  the  cross-correlation  of  the  input  x  and  the  stored  image  xm ,  while 
^2^=1  Xj  is  the  dc  level  of  the  signal  x,  and  am  is  the  dc  level  of  the  stored  image  xm.  The 
subtraction  of  the  product  of  the  dc  levels  from  the  correlation  signal  gives  the  weight. 
Thus,  only  the  pixels  that  have  correlation  with  the  stored  xm  sufficiently  larger  than  the  dc 
levels  contribute  a  significant  component  to  y.  Therefore,  only  the  strongest  stored  image 
component  that  appears  in  x  will  be  enhanced  in  the  feedback,  while  weaker  components 
are  suppressed  by  subtracting  the  average  image.  This  argument  suggests  that  the  loop 
will  work  as  an  associative  memory. 

In  the  case  of  the  optical  loop,  the  neural  gain  is  the  main  factor  that  determines 
whether  the  loop  can  recall  a  correct  memory  state.  The  modeled  network  shows  the  same 
behavior.  To  illustrate  this  point,  we  perform  simulations  with  different  gains.  Fig.  26 
shows  an  example  of  computer  simulation.  There  are  three  patterns  stored  in  the  high- 
pass  network;  each  pattern  has  a  30  pixels.  In  the  figure,  the  height  at  each  position 
represents  the  signal  strength  of  that  pixel.  When  a  partial  x2  is  input  into  the  system, 
it  converges  to  a  stable  state  that  is  closest  to  the  x2  after  160  iterations.  As  the  gain 
is  reduced,  the  loop  takes  more  iterations  to  reach  the  steady  state.  However,  when  the 
gain  is  reduced  to  below  a  certain  level,  the  loop  can  no  longer  sustain  the  loop  signal  and 
the  signal  gradually  decays  to  zero.  This  is  illustrated  in  Fig.  27.  On  the  other  hand,  as 
the  gain  is  increased,  the  loop  evolves  to  a  steady  state  in  fewer  steps.  But  if  the  gain  is 
too  high,  the  loop  converges  to  a  mixed  state  of  the  stored  images,  which  may  be  very 
distorted  and  unrecognizable.  Fig.  28  shows  simulations  under  high-gain  conditions.  The 
results  of  these  simulations  show  that  the  modeled  network  has  the  same  characteristics 
as  the  optical  loop:  It  has  the  capability  of  recognizing  distorted  images,  but  there  is  a 
trade-off  between  distortion  tolerance  and  discrimination  capability,  which  is  governed  by 
the  neural  gains. 

Note  that  in  the  simulation  of  Fig.  26  the  steady-state  image  of  the  loop  is  slightly 
different  from  the  original  images.  But  we  still  consider  it  a  correct  recall.  Here  the 
correct  memory  means  that  it  has  a  similar  shape  as  the  original  image,  although  there  is 
a  slight  distortion  and  the  signal  level  is  weaker.  The  reason  for  that  distortion  is  easily 
understandable  by  inspecting  Eq.  27.  We  see  that  the  feedback  contains  not  only  the 
autocorrelation  signal,  but  also  the  cross-correlations  with  all  other  memories.  Depending 
on  the  values  of  the  cross-correlation  and  the  gain  function,  some  of  the  crosstalks  will  be 
detected  by  the  neurons  and  survived  in  the  loop;  thus,  the  total  output  is  distorted.  The 
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Fig.  27:  The  dynamics  of  the  high-pass  loop  with  a  low  gain. 
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higher  the  neural  gain  the  worse  the  distortions.  We  will  return  to  this  point  later  when 
we  examine  the  steady-state  solution  of  the  dynamic  equations  of  the  system. 

An  interesting  question  that  we  want  to  address  is  whether  the  high-pass  hologram  is 
necessary  and  how  it  affects  the  system  performance.  Suppose  the  first  hologram  is  not  a 
high-pass  version,  then  the  interconnection  strength  can  be  obtained  by  setting  am  =  0  in 
Eq.  24.  We  get 

Af 

<■>«  =  £  (30) 

m=l 

Similarly,  the  feedback  signal  can  be  obtained  by  setting  am  =  0  in  Eq.  28, 

M 

y  =J2  (x  ■  xm)xm 

m=  1 

The  feedback  signal  now  is  simply  the  superposition  of  the  stored  images  each  being 
weighted  by  the  cross-correlation  of  the  input  image  and  the  stored  images.  The  strongest 
correlation  gives  the  strongest  feedback  signal  and  it  determines  the  state  to  which  the 
loop  will  most  possibly  converge.  Hence  the  loop  still  works  as  an  associative  memory. 
However,  because  there  is  no  mechanism  to  suppress  the  crosstalks,  noise  is  also  easily 
picked  up  by  the  neurons.  The  discrimination  capability  of  the  loop  will  be  poorer  than 
the  high-pass  loop.  Fig.  29  shows  an  example  of  the  computer  simulation.  The  images 
used  in  the  high-pass  simulations  are  used  here.  Simulations  show  that  as  long  as  the 
stored  images  have  crosstalks,  then  the  low-pass  loop  always  converges  to  a  mixed  state. 
There  is  only  a  very  narrow  range  of  the  gain  where  the  system  gives  the  correct  output. 
Therefore,  the  high-pass  hologram  is  very  crucial  in  the  performance  of  the  loop. 

To  this  point,  we  have  developed  a  neural  network  model  of  the  optical  associative 
loop,  and  we  have  investigated  its  characteristics  with  computer  simulations.  Results  show 
that  the  model  matches  well  with  the  optical  loop.  However,  the  above  treatment  is  not 
very  rigorous  and  does  not  show  the  dynamic  behavior  of  the  system.  Therefore,  it  is  not 
completely  satisfactory.  In  general,  only  the  solution  of  the  dynamic  equations  can  fulfill 
our  purpose. 

In  our  system,  an  optical  neuron  is  simulated  by  one  pixel  of  the  LCLV,  which  gives 
response  x,  to  its  input  y*.  The  dynamics  of  the  neuron  is  described  by  the  following 
equation 

~  =  -n  +g(yi),  i  =  l...N,  (32) 

where  g(y, )  is  a  nonlinear  function  describing  the  neuron  response.  To  complete  the  feed¬ 
back  loop,  we  substitute  the  expression  of  Eq.  26  into  Eq.  32,  we  obtain 


(31) 


(33) 


where  tui;  is  the  high-pass  version  of  the  interconnection  strength  given  by  Eq.  24.  This  is 
the  equation  describing  the  dynamics  of  the  optical  loop.  In  what  follows,  we  will  discuss 
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the  system’s,  dynamics  using  a  geometrical  method  [20].  The  main  idea  of  the  method  is 
that  corresponding  to  each  set  of  stored  memories  we  first  define  a  new  vector  space;  then 
the  dynamic  equations  are  transformed  consequently  into  this  space  and  the  dynamics 
can  be  observed  as  a  phase  flow  on  the  hyper-surface.  We  assume  that  the  stored  images 
x1  . .  .  xM  are  linearly  independent.  This  is  reasonable  since  the  number  of  pixels  in  the 
image,  N,  is  usually  much  larger  than  M,  and  we  are  not  considering  the  trivial  case  where 
one  of  the  stored  images  is  a  linear  combination  of  the  others.  Since  the  stored  images  are 
linearly  independent,  they  span  a  vector  space  Vi.  Then,  we  can  form  a  vector  space  V2 
that  is  orthogonal  to  Vi  such  that  the  N-dimensional  vector  space  is  the  direct  sum 
of  the  two  sub- vector  spaces;  i.e., 

Rn  =  Vx  ©  V2  (34) 

It  is  shown  in  the  Appendix  that  we  can  derive  a  reciprocal  basis  =  {y1 . .  .  yM}  from 
{x1  . . .  xM  }  for  V\ ,  and  an  orthonormal  basis  02  =  {y Jw+1 . . .  yN  }  for  V2  such  that  0i  U/?2  = 
0  forms  a  basis  for  R'v.  Then  any  vector  in  RA  can  be  expressed  as  a  linear  combination 
of  y1,  /  =  1, . . .  N. 

It  is  shown  in  the  appendix  that  the  N-coupled  differential  equations  of  Eq.  33  can  be 
represented  in  terms  of  the  0  coefficients  as 


dxi 

dt 


-x  i  +  g 


( t  '■"*") 

'  m=l  ' 


(35) 


Note  that  there  are  TV-coupled  differential  equations,  but  there  are  only  M  parameters  of 
cm  for  all  the  equations.  It  is  also  derived  in  the  appendix  that  Eq.  35  in  turn  can  be 
transformed  into  two  sets  of  equations 


^  =  -c<  +  Y(Xi‘  -  *1)9  ( Y 

»=  1  '  m=  1  ' 

A  N  /  M  \ 

=  -ci+YW  -b>) 9 1 Y  c™i'ra 

i~  1  A  m=l  ' 


1  =  1,. ..,M, 
l  =  M  +  1, . . . ,  N 


(36) 

(37) 


where 

1  N 

=  (38) 

J=1 

Several  conunents  can  be  made  on  Eq.  36  and  37.  First,  we  see  that  the  components  cj , 
. . .,  cm  are  coupled  together,  but  they  are  de-coupled  from  the  components  cm+ 1,  •  •  c^. 

On  the  other  hand,  the  driving  forces  for  cm+\,  •  •  •,  depend  only  on  cx,  . . .,  cm  ■  This 
means  that  the  dynamics  of  the  system  are  completely  determined  by  c\ ,  . . .,  cm,  although 
cm+ 1,  •••,  cjv  are  not  necessarily  zero.  Therefore,  we  only  need  to  study  the  dynamics 
of  these  M  components.  The  equilibrium  states  can  be  obtained  from  Eq.  35,  by  letting 
d/dt  —  0,  we  get 

Xi  ~  9(  Y  (39) 

'  m=l  ' 
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We  see  that  the  equilibrium  states  are  indeed  determined  by  cj,  . . cm-  Each  of  these 
cm  represents  the  component  of  the  high-pass  version  of  the  input  vector  on  the  reciprocal 
vectors  of  the  m*h  stored  image,  which  in  some  sense  is  proportional  to  the  correlation 
between  x  and  xm.  The  stored  images  which  have  stronger  correlation  give  stronger 
components  to  the  equilibrium  image.  Note  that  if  the  neurons  are  sensitive  enough,  they 
will  also  pick  up  weak  cross-correlation  components.  Therefore,  the  output  image  will  not 
be  exactly  the  same  as  the  stored  images.  However,  the  summation  of  all  components  from 
the  stored  images  is  thresholded  by  the  neurons.  Thus,  the  nonlinear  gain  function  provides 
an  enhancement  of  the  closest  stored  image.  This  effect  helps  the  network  converge  to  the 
state  near  the  stored  images. 

In  principle,  the  dynamics  of  the  system  and  its  convergence  properties  can  be  obtained 
by  solving  Eq.s  36  and  37.  In  practice,  it  is  impossible  to  obtain  explicit  solutions.  In  what 
follows  we  present  a  geometrical  method  to  illustrate  how  the  system  evolves  to  a  stable 
state,  and  how  it  is  influenced  by  the  parameters  such  as  gain  and  initial  conditionr.  In 
order  to  illustrate  the  concept,  we  will  consider  the  case  where  only  two  images,  x1  and 
x2,  me  stored  in  the  memory.  As  we  shall  see,  the  two-image  case  contains  all  the  salient 
features  of  the  dynamics.  As  discussed  in  the  previous  paragraph,  we  need  only  to  solve 
two  dynamic  equations  in  the  two-image  case.  By  Eq.  36,  we  have 

,  N 

-TT  =  -Cl  +S(X*'  ~aMClXi  +C2*i)  (40) 

i=l 

.  N 

-j-  =  ~c2  +  -  a2)g(clx}  +  c2x 2).  (41) 

1=  1 

Recall  that  aj  and  a2  are  the  average  levels  of  the  input  images  x1  and  x2.  Let  hi(ci,c2) 
represent  the  summation  term  in  Eq.  40,  and  h2(c\,c2)  the  summation  term  in  Eq.  41. 
These  two  terms  axe  the  driving  force  for  c\  and  c2 ,  respectively.  We  now  discuss  the 
dynamics  of  the  two  images.  For  simplicity,  assume  that  x1  and  x2  have  no  overlapping 
nonzero  components.  An  example  is  stiown  in  Fig.  30. 

It  is  seen  that  x\  can  be  nonzero  only  when  x\  =  0,  and  vice  versa.  In  this  case,  the 
driving  forces  can  be  written  as 


N 

hi(cuc2)  =  53  (x|  -  aOfftctxj)  -  ax  53  0(c 2®?)  (42) 

xj#0  *?*0 

N 

h2{ci,c2)=  53  (x?  ”  a2)ff(c2X?)  -  a2  ^  g(clXj)  (43) 

x?*0  o 

There  are  two  terms  in  each  of  the  driving  forces.  Consider  hi  (ci ,  c2 ).  The  first  term  comes 
from  the  correlation  between  the  neuron  state  ^(cix1)  and  the  stored  image  (x1  —  dj),  and 
the  second  term  results  from  the  coupling  between  cj  and  c2  through  the  dc  level  a\. 
Since  aj  and  the  gain  function  g(x )  are  always  positive,  the  second  term  gives  a  negative 
contribution  to  the  driving  force.  This  means  that  the  coupling  pulls  the  system  away 
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from  x1.  The  same  description  also  applies  to  c 2.  We  plot  &i(ci,c2)  against  cj  for  C2  =  0 
and  C2  7^  0  in  Fig.  31(a). 

In  the  figure,  the  solid  curve  represents  the  case  where  C2  =  0,  and  the  dashed  curve 
represents  the  case  for  c2  7^  0.  We  also  plot  the  line  h(c\)  =  c\  in  the  same  figure.  It  is 
seen  that  there  are  three  intersections,  P,  Q,  and  R,  between  the  straight  line  and  the 
solid  curve.  The  plane  is  divided  into  four  regions.  In  regions  1  and  3,  c\  is  smaller  than 
h\(c\ ,  C2 )  and  >  0.  Thus,  in  these  regions  the  system  state  evolves  in  the  direction 
of  increasing  cj .  This  is  represented  by  the  arrow  pointing  to  the  right  in  the  figure.  On 
the  other  hand,  in  regions  2  and  4,  <  0;  thus,  the  system  evolves  toward  decreasing 

cj.  It  can  be  seen  that  the  points  P  and  R  are  stable  points,  and  Q  is  a  saddle  point. 
Now  suppose  we  increase  C2.  Then  g(c 2x2)  increases,  and  hi(ci,c2)  decreases.  It  can  be 
seen  from  Fig.  31(a)  that  points  Q  and  R  move  towards  each  other.  The  two  points  merge 
into  one  point  at  large  values  of  C2.  On  the  other  hand,  the  point  P  moves  downward 
as  c2  increases.  If  we  plot  the  trajectories  of  P,  Q,  and  R  for  the  coefficient  Cj  in  the 
(c  1,  c2)  plane,  we  obtain  Fig.  31(b).  Since  the  gain  function  g(x)  is  positive  and  symmetric 
with  respect  to  r,  the  trajectories  are  symmetric  with  respect  to  the  C2  axis.  The  above 
argument  shows  how  the  dynamical  behavior  of  the  system  can  be  understood  using  a 
geometrical  method. 

By  going  through  the  same  procedure,  we  can  also  obtain  the  trajectories  leading  to 
the  equilibrium  points  of  c2.  We  plot  the  two  groups  of  trajectories  in  the  same  (ci,c2) 
plane.  Fig.  32  shows  the  result.  We  see  that  there  are  7  equilibrium  points:  one  source, 
three  sinks,  and  three  saddles.  The  three  sinks  represent  the  null  state  (no  image)  and  the 
two  stored  images.  Point  1  represents  the  stable  state  corresponding  to  stored  image  x1, 
since  at  that  position  c i  is  large  and  c2  is  small.  On  the  other  hand,  at  point  2  C\  is  small 
and  c2  is  large.  This  represents  the  stable  state  corresponding  to  stored  image  x2  It  can 
be  seen  from  the  figure  that  if  we  start  from  an  initial  state,  which  is  close  to  one  of  the 
stored  states,  then  the  system  will  converge  to  that  state.  Otherwise,  it  will  decay  to  zero. 

From  the  geometrical  diagram  we  see  that  the  stable  state  is  always  a  mixed  state  of 
the  stored  memories.  The  extent  of  mixture  can  be  reduced  by  reducing  the  neural  gain. 
However,  if  the  gain  is  too  small,  then  the  system  will  not  be  able  to  sustain  the  stored 
memories.  As  shown  in  Fig.  33(a),  when  the  gain  is  very  low,  there  is  only  one  intersection 
point  O  between  the  line  h(c)  =  c  and  the  curve  of  the  driving  force.  The  corresponding 
trajectories  leading  to  O  are  drawn  in  Fig.  33(b).  It  is  seen  that  the  only  equilibrium  state 
is  the  null  state  at  P.  No  matter  where  the  initial  state  is,  the  system  always  decays  to 
zero. 

On  the  other  hand,  suppose  the  neural  gain  is  very  high.  The  trajectories  to  P,  Q, 
and  R  axe  shown  in  Fig.  34. 

We  see  that  there  Eire  two  more  equilibrium  points  them  the  case  shown  in  Fig.  32:  a 
stable  point  m  and  a  saddle  point  s.  The  state  m  is  a  strongly  mixed  state  of  x1  and  x2. 
We  also  see  that  m  has  a  large  region  of  attraction.  Thus,  we  cam  not  set  too  high  a  gain. 

Next  consider  the  case  where  the  stored  memories  have  some  slight  overlap.  The  shape 
and  position  of  the  trajectory  lines  will  be  altered  somewhat.  Since  the  neural  function  is 
continuous,  so  is  the  driving  force.  Thus,  the  general  features  of  the  system  will  be  the 
same.  However,  eis  the  overlapping  between  the  stored  states  increases  more  amd  more, 
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Fig.  31:  The  Driving  Force  and  the  Dynamics  of  the  Loop,  (a)  The  driving  force  for 
the  first  stored  image,  (b)  The  trajectories  of  the  equilibrium  states  of  the  first  image. 
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Fig.  32:  The  phase  flow  of  the  two-image  auto-associative  memory.  States  O,  1,  2  are 
stable.  States  3,  4,  5  are  unstable  (saddle  points).  State  6:  Source  state,  (unstable) 
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Pig.  33:  The  Dynamics  of  the  Loop  at  Low  Gain,  (a)  The  driving  force  for  the  firs 
d  image,  (b)  Trajectories  for  the  equilibrium  points.  Point  P  is  the  only  stable  stat< 
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the  trajectories  of  the  equilibrium  points  will  be  more  and  more  distorted.  As  a  result,  the 
stable  points  that  give  the  stored  memory  states  will  have  more  “mixing”  with  the  other 
stored  state.  Thus,  we  would  not  expect  the  system  to  work  well  in  the  case  of  strong 
overlapping. 

We  can  now  investigate  the  dynamics  of  the  system  for  the  all-pass  hologram  in  the 
memory  loop.  If  we  do  not  make  the  dc-level  subtraction  in  storing  the  memories,  then 
Equations  40  and  41  become 


dc\ 

dt 

dC2 

dt 


N 

=  -C!  +  ^2x)g(Clx}  +C2x?) 
1=1 
N 

=  ~C2  +  ^2  +  C2*?)- 
»=1 


(44) 

(45) 


By  going  through  similar  arguments,  we  can  draw  the  trajectories  leading  to  the  equilib¬ 
rium  points  of  the  system.  Fig.  35  shows  one  example. 

It  is  seen  that  there  axe  four  stable  states:  two  memory  states,  m i  and  m2,  one  null 
state  ,  and  one  mixed  state  P.  If  we  decrease  the  neural  gain,  then  the  points  m j,  m2,  Q, 
and  R  may  disappear.  However,  the  mixed  state  P  always  exists.  Therefore,  the  high-pass 
hologram  is  crucial  for  good  performance  of  the  memory  loop. 

7  Conclusion 

In  this  report  we  have  successfully  implemented  an  optical  loop  of  the  Hopfield-type 
neural  network.  The  loop  consists  of  a  single  layer  of  neurons  with  feedback.  The  2-D 
array  of  neurons  are  simulated  by  an  LCLV.  The  interconnections  between  the  neurons  are 
implemented  by  using  planar  holograms. 

Experimental  results  show  that  the  loop  performs  the  function  of  auto-association 
which  recognizes  2-D  images  from  a  distorted  input.  The  distortions  may  come  from 
rotating,  scaling,  shifting,  partial  blocking,  or  erroneous  input.  As  long  as  the  input 
provides  enough  information,  the  loop  always  converges  to  a  stable  state  corresponding 
to  the  correct  stored  image.  The  similarity  between  the  input  and  the  stored  images 
determines  how  fast  the  loop  converges. 

The  gain  function  is  another  factor  that  determines  the  dynamics  of  the  loop.  Too 
low  gain  cannot  sustain  a  stable  memory.  High  gain  provides  more  tolerance  to  the  system 
to  recognize  a  distorted  image.  However,  too  high  gain  also  induces  mixed  states  such 
that  the  loop  makes  incorrect  recognitions.  The  trade-off  between  distortion  tolerance  and 
discrimination  capability  is  set  by  the  gain  function,  and  the  value  of  the  optimum  gain 
depends  on  the  specific  images  stored. 

We  have  presented  a  neural  network  model  for  the  optical  loop.  Dynamic  equations 
for  the  network  are  formulated.  Since  the  optical  neurons  respond  to  light  intensities,  the 
input  vs.  output  relationship  of  the  neurons  is  not  a  simple  sigmoid  function.  Rather,  it 
is  a  sigmoid  function  symmetrical  for  positive  and  negative  inputs.  Thus,  the  slope  of  the 
neuron  function  is  not  always  positive,  and  the  conventional  proof  of  convergence  does 
not  apply  here.  Although  we  have  not  found  an  energy  function  for  our  loop,  we  have 
presented  a  geometrical  method  for  inspecting  dynamic  behaviors  of  the  loop.  The  flow  of 
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the  state  in  the  phase  diagram  helps  us  visualize  the  convergence  properties  of  the  loop. 
The  diagram  shows  that  the  stable  state  is  always  a  mixed  state  of  the  stored  images,  the 
degree  of  mixture  depending  on  the  gain  function. 
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Appendix 

Dynamic  Equations  for  the  Optical  Associative  memory 

This  appendix  will  focus  on  the  derivation  of  the  dynamic  equations  for  the  optical 
associative  memory  loop  described  in  this  report.  The  schematic  diagram  of  the  optical 
loop  is  shown  in  Fig.  5.  The  main  idea  is  that  corresponding  to  the  set  of  stored  images, 
we  can  construct  a  vector  space  RN,  where  N  is  the  number  of  neurons  of  the  system. 
Then  the  N-coupled  dynamic  equations  of  the  neurons  can  be  expressed  in  terms  of  the 
basis  coefficients  in  RN.  This  transformation  simplifies  the  form  of  dynamic  equations  so 
that  the  dynamic  properties  of  the  system  can  be  inspected  using  a  geometrical  method. 

A.  Vector  Space  RN  and  the  Basis 

We  assume  that  the  stored  images  xx,x2, . . .  ,xM  axe  linearly  independent.  We  then 
decompose  the  vector  space  RN  into  two  subspaces  Vi  and  V2,  where  Vj  is  the  vector 
space  spanned  by  the  stored  images  and  V2  is  normal  to  Vj;  i.e., 

RN  =  V]  ®  V2,  (46) 

Vi  =  span{x\x2,...,xM},  (47) 

V2  =  {y|x-y  =  0,Vx  €  Vi),  (48) 

where  ©  is  the  direct  sum  of  vector  spaces.  We  now  define  a  reciprocal  basis  0X  = 
{y\y2, . .  ■  ,yM)  for  V,,  such  that 

y‘  xi  =  6ij  i,j  =  1, . . . ,  AT.  (49) 

Next  we  select  an  orthonormal  basis  02  —  (yM+1,  •  •  •  ,  yN}  for  V2.  We  then  have 

0  =  01  U  02 

=  {yV-.yN}  (so) 

which  forms  a  basis  for  RN.  Thus,  any  image  in  RN  can  be  expressed  in  terms  of  the 
basis  0: 

N 

*  =  £‘Wrk-  (51) 

k=  1 


B.  Dynamic  Equations  for  the  Optical  Loop 


It  was  shown  in  Section  6  that  the  dynamic  equation  for  the  neurons  in  the  optical 
loop  is 


di{ 

dt 


i  =  1 ...  JV 


(52) 
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where  Wij  is  the  high-pass  version  of  the  interconnection  strength  given  by  Eq.  24.  By 
Eq.  24  and  25,  we  have 


= Y.  (£(*” -<■»<)*!")*, 

j~  1  j=  1  V  m=l  7 

N  /  M  v  M  N 

= £  ( £  *r*r)*i  -  £ 5Za«»*r*; 

j=  1  '  m=l  7  m=l  j=l 

N  /  M  \  /  N  s  /  M  W  v 

=E  E-r.? >/-  EEfr-r 

>=1  x  m=l  7  xt=l  7  '  m=l  j=l  7 


=±(±*?*7)(*,- i£4 

J=1  Vm  =  l  7  v  fc=  1  7 


Note  that  is  the  component  of  the  vector  Ylk=i  x*)  Since 

is  also  a  vector  in  RN  and  any  vector  in  RN  can  be  expanded  in  the 

basis  /?,  we  let 


x  k=i  7  *:=i 


The  feedback  signal  in  the  loop  then  becomes 


N  ,  M  \  /  IV  \ 

Z  =  x?x?)  ( Y,  c*k?) 

j=l  j=  1  V  m=l  7  X  i=l  7 

M  N  y  N  v 

=  ££(£*rv‘W 

m=l  *=1  x  >=1  7 


M  y  N  V 

=  H  W- 

m=l  x*=l  7 


By  applying  the  orthnonnal  property  of  the  reciprocal  vectors  xm  and  ym  in  the  above 
equation,  we  obtain 

N  M 

y  ]  wjjXj  =  y  ]  cmxt .  (56) 

j=l  m=l 


The  dynamic  equation  then  becomes 


dii 

ir  =  -Ii+5 


/  M  v 

(£Cmir) 


*  =  1 ...  AT. 
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Note  that  cm  is  the  projection  of  the  high-pass  version  of  the  input  on  the  stored  vector 
xm .  It  is  seen  that  the  system  dynamics  is  specified  by  the  M  coefficients  of  the  reciprocal 
vectors  of  the  stored  images.  In  what  follows  we  will  derive  the  equation  of  motion  for 
these  coefficients. 

Multiplying  both  sides  of  Eq.  57  by  1  /N  and  summing  over  i,  we  get 
N  ,  ,  N  i  N  /  M 

i  =  1 ...  IV.  (58) 


t=l  i=l  i=l  '  m=l  7 


Subtracting  Eq.  58  from  Eq.  57  gives 

N  „  N 


ji=l  7=1  '  m=l  7  i=l  x'm=l  7 


•  (59) 


Note  that  the  terms  in  the  bracket  can  expanded  in  the  basis  thus,  the  equation  of 
motion  is  given  in  terms  of  the  0  coefficients  as 


(60) 


(61) 


N  .  N  ,  M  .  -  N  ,  M  . 

E  + »( E  «-*r)  -  £  E»(  E  «.»r). 

k=l  0  fc=l  '  m=l  7  1  1=1  '  m=l  7 

Multiplying  both  sides  of  the  above  equation  by  x[  and  then  summing  over  i  gives 
N  ,  N  N  ,  M  N 

~^(xl  •  y1)  =  -  5Z  c^x‘  •  y1)  +  £ ( J2  cm*r )  , 

*=1  Jfc=l  »=1  '  m=l  7 

V  i=l  7  i=l  Vm=l  7 

Applying  the  orthonormal  relation  between  the  stored  images  and  the  reciprocal  basis,  we 
obtain 

d  N  /  M  \ 

=  _C'  +  X^Z«'  _a<M  C^XT)  /=1,...,M.  (62) 

i=l  '  m=l  7 

Recall  that  a/  is  the  average  level  of  the  stored  image  x*.  Similarly,  the  equation  of  motion 
for  ca/+i,  •  •  • ,  cat  can  be  obtained  by  multiplying  both  sides  of  Eq.  60  by  y-,  and  summing 
over  t.  We  get 

N  /  M  N 

l  =  M  +  1,. . .  ,N,  (63) 


dci 

dt 


/  \ 
=  -ci  +  -  bt)g  f  ^  ) 

i=l  'm=l  7 


where  6j  is  defined  as 


1  N 

l,  =  ffE»i 


(64) 


i=l 


The  dynamic  behavior  of  the  optical  loop  is  thus  completely  determined  by  Equ.s  63  and  64. 
Instead  of  solving  these  equations,  we  have  presented  a  geometrical  method  in  Equ.stion 
6  for  investigating  the  convergence  properties  of  the  loop  and  the  important  parameters 
that  affect  the  dynamics. 
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